{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started: Fashion Mnist on Hops Notebook\n",
    "---\n",
    "\n",
    "<font color='red'> <h3>Tested with TensorFlow 1.10</h3></font>\n",
    "\n",
    "In this notebook we are going to master running Deep Learning code on Hops using Jupyter notebooks and HopsFS, the highly efficient distributed file system that Hops provides. HopsFS is a fork of Apache HDFS and operations for reading and writing files to HopsFS is the same as with HDFS that TensorFlow supports.\n",
    "For more information about how to read from HDFS, please see: https://www.tensorflow.org/deploy/hadoop\n",
    "\n",
    "In this example program we are going to:\n",
    "- Run Fashion Mnist code using `tf.contrib.learn.Experiment` and `tf.estimator.Estimator`\n",
    "- Using the hops python library to run TensorFlow on the Hops Platform\n",
    "- Read data from a Dataset which is located in your project, on HopsFS (HDFS)\n",
    "- Define a convolutional neural network\n",
    "- Search for best hyperparameters given some search space\n",
    "- Run training for each combination of hyperparameters using the Experiment API\n",
    "- Monitor training using TensorBoard to see accuracy and loss for every hyperparameter configuration\n",
    "- Monitoring training by looking at logs\n",
    "- Find where the model and TensorBoard events for each hyperparameter configuration is saved in your project\n",
    "- Visualize TensorBoard events from the previous TensorBoard run\n",
    "\n",
    "## Table of contents:\n",
    "\n",
    "### [TensorFlow on Hops](#paradigm)\n",
    "### [Fashion Mnist training on Hops](#mnist)\n",
    "### [Hyperparameter search](#hyperparam)\n",
    "### [Launching TensorFlow jobs](#starting)\n",
    "### [Monitoring execution - TensorBoard](#tensorboard)\n",
    "### [Monitoring execution - Logs](#logs)\n",
    "### [Increasing throughput - Running jobs in parallel](#parallel)\n",
    "### [Show TensorBoard from previous runs](#visualize)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hops TensorFlow programming paradigm <a class=\"anchor\" id='paradigm'></a>\n",
    "\n",
    "To be able to run your TensorFlow code on Hops, the code for the whole program needs to be provided and put inside a wrapper function. Everything, from importing libraries to reading data and defining the model and running the program needs to be put inside a wrapper function.\n",
    "\n",
    "You can also submit one or more `.py`, `.zip` or `.egg` files that contain your code and import them in the wrapper function. To include files, navigate back to HopsWorks and restart restart Jupyter, you can then include files in the Jupyter configuration.\n",
    "\n",
    "## The `hops` python module\n",
    "\n",
    "Below you can see the aforementioned wrapper function, which is coincidently named `wrapper` but could potentially be named anything. You can see two imports from the `hops` module, a `tensorboard` and an `hdfs` module. These are the only two modules that you will need to use in your TensorFlow wrapper function. \n",
    "\n",
    "### Using the `tensorboard` module\n",
    "The `tensorboard` module allow us to get the log directory for summaries and checkpoints to be written to the TensorBoard we will see in a bit. The only function that we currently need to call is `tensorboard.logdir()`, which returns the path to the TensorBoard log directory. Furthermore, the content of this directory will be put in as a Dataset in your project in HopsFS after each hyperparameter configuration is finished. The `experiment.launch` function, that we will look at abit further down will return the exact path, which you can then navigate to using HopsWorks to inspect the files.\n",
    "\n",
    "The directory could in practice be used to store other data that should be accessible after each hyperparameter configuration is finished.\n",
    "```python\n",
    "# Use this module to get the TensorBoard logdir\n",
    "from hops import tensorboard\n",
    "tensorboard_logdir = tensorboard.logdir()\n",
    "```\n",
    "\n",
    "\n",
    "### Using the `hdfs` module\n",
    "The `hdfs` module provides a single method to get the path in HopsFS where your data is stored, namely by calling `hdfs.project_path()`. The path resolves to the root path for your project, which is the view that you see when you click `Data Sets` in HopsWorks. To point where your actual data resides in the project you to append the full path from there to your Dataset. For example if you create a mnist folder in your Resources Dataset, which is created automatically for each project, the path to the mnist data would be `hdfs.project_path() + 'Resources/mnist`\n",
    "```python\n",
    "# Use this module to get the path to your project in HopsFS, then append the path to your Dataset in your project\n",
    "from hops import hdfs\n",
    "project_path = hdfs.project_path()\n",
    "```\n",
    "\n",
    "![image11-Dataset-ProjectPath.png](../../../images/datasets.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Fashion Mnist on Hops <a class=\"anchor\" id='cifar10'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The wrapper function\n",
    "\n",
    "Here we define the aforementioned wrapper function containing the code to run, with the hyperparameter arguments `learning_rate` and `dropout`. It simply contains all the TensorFlow code that we want to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wrapper(learning_rate, dropout):\n",
    "\n",
    "    import tensorflow as tf\n",
    "    import numpy as np\n",
    "    from hops import tensorboard\n",
    "    from hops import hdfs\n",
    "\n",
    "    # Training Parameters\n",
    "    num_steps = 20\n",
    "    batch_size = 128\n",
    "\n",
    "    # Network Parameters\n",
    "    num_input = 784 # MNIST data input (img shape: 28*28)\n",
    "    num_classes = 10 # MNIST total classes (0-9 digits)\n",
    "\n",
    "    train_filenames = [hdfs.project_path() + \"TourData/mnist/train/train.tfrecords\"]\n",
    "    validation_filenames = [hdfs.project_path() + \"TourData/mnist/validation/validation.tfrecords\"]\n",
    "\n",
    "    # Create the neural network\n",
    "    # TF Estimator input is a dict, in case of multiple inputs\n",
    "    def conv_net(x, n_classes, dropout, reuse, is_training):\n",
    "\n",
    "        # Define a scope for reusing the variables\n",
    "        with tf.variable_scope('ConvNet', reuse=reuse):\n",
    "\n",
    "            # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n",
    "            # Reshape to match picture format [Height x Width x Channel]\n",
    "            # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n",
    "            x = tf.reshape(x, shape=[-1, 28, 28, 1])\n",
    "\n",
    "            # Convolution Layer with 32 filters and a kernel size of 5\n",
    "            conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)\n",
    "            # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
    "            conv1 = tf.layers.max_pooling2d(conv1, 2, 2)\n",
    "\n",
    "            # Convolution Layer with 32 filters and a kernel size of 5\n",
    "            conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu)\n",
    "            # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n",
    "            conv2 = tf.layers.max_pooling2d(conv2, 2, 2)\n",
    "\n",
    "            # Flatten the data to a 1-D vector for the fully connected layer\n",
    "            fc1 = tf.contrib.layers.flatten(conv2)\n",
    "\n",
    "            # Fully connected layer (in tf contrib folder for now)\n",
    "            fc1 = tf.layers.dense(fc1, 1024)\n",
    "            # Apply Dropout (if is_training is False, dropout is not applied)\n",
    "            fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)\n",
    "\n",
    "            # Output layer, class prediction\n",
    "            out = tf.layers.dense(fc1, n_classes)\n",
    "\n",
    "        return out\n",
    "\n",
    "\n",
    "    # Define the model function (following TF Estimator Template)\n",
    "    def model_fn(features, labels, mode, params):\n",
    "\n",
    "        # Build the neural network\n",
    "        # Because Dropout have different behavior at training and prediction time, we\n",
    "        # need to create 2 distinct computation graphs that still share the same weights.\n",
    "        logits_train = conv_net(features, num_classes, dropout, reuse=False, is_training=True)\n",
    "        logits_test = conv_net(features, num_classes, dropout, reuse=True, is_training=False)\n",
    "\n",
    "        # Predictions\n",
    "        pred_classes = tf.argmax(logits_test, axis=1)\n",
    "        pred_probas = tf.nn.softmax(logits_test)\n",
    "\n",
    "        # If prediction mode, early return\n",
    "        if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "            return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)\n",
    "\n",
    "        # Define loss and optimizer\n",
    "        loss_op = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_train, \n",
    "                                                                                labels=tf.cast(labels, dtype=tf.int32)))\n",
    "        \n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "        train_op = optimizer.minimize(loss_op, global_step=tf.train.get_global_step())\n",
    "\n",
    "        # Evaluate the accuracy of the model\n",
    "        acc_op = tf.metrics.accuracy(labels=labels, predictions=pred_classes)\n",
    "\n",
    "        image = tf.reshape(features[:10], [-1, 28, 28, 1])\n",
    "        tf.summary.image(\"image\", image)\n",
    "\n",
    "        # TF Estimators requires to return a EstimatorSpec, that specify\n",
    "        # the different ops for training, evaluating, ...\n",
    "        estim_specs = tf.estimator.EstimatorSpec(\n",
    "          mode=mode,\n",
    "          predictions=pred_classes,\n",
    "          loss=loss_op,\n",
    "          train_op=train_op,\n",
    "          eval_metric_ops={'accuracy': acc_op})\n",
    "\n",
    "        return estim_specs\n",
    "\n",
    "\n",
    "    def data_input_fn(filenames, batch_size=128, shuffle=False, repeat=None):\n",
    "\n",
    "        def parser(serialized_example):\n",
    "            \"\"\"Parses a single tf.Example into image and label tensors.\"\"\"\n",
    "            features = tf.parse_single_example(\n",
    "                serialized_example,\n",
    "                features={\n",
    "                    'image_raw': tf.FixedLenFeature([], tf.string),\n",
    "                    'label': tf.FixedLenFeature([], tf.int64),\n",
    "                })\n",
    "            image = tf.decode_raw(features['image_raw'], tf.uint8)\n",
    "            image.set_shape([28 * 28])\n",
    "\n",
    "            # Normalize the values of the image from the range [0, 255] to [-0.5, 0.5]\n",
    "            image = tf.cast(image, tf.float32) / 255 - 0.5\n",
    "            label = tf.cast(features['label'], tf.int32)\n",
    "            return image, label\n",
    "\n",
    "        def _input_fn():\n",
    "            # Import MNIST data\n",
    "            dataset = tf.data.TFRecordDataset(filenames)\n",
    "\n",
    "            # Map the parser over dataset, and batch results by up to batch_size\n",
    "            dataset = dataset.map(parser)\n",
    "            if shuffle:\n",
    "                dataset = dataset.shuffle(buffer_size=128)\n",
    "            dataset = dataset.batch(batch_size)\n",
    "            dataset = dataset.repeat(repeat)\n",
    "            iterator = dataset.make_one_shot_iterator()\n",
    "\n",
    "            features, labels = iterator.get_next()\n",
    "\n",
    "            return features, labels\n",
    "\n",
    "        return _input_fn\n",
    "\n",
    "\n",
    "    run_config = tf.contrib.learn.RunConfig(\n",
    "        model_dir=tensorboard.logdir(),\n",
    "        save_checkpoints_steps=5,\n",
    "        save_summary_steps=5,\n",
    "        log_step_count_steps=5)\n",
    "\n",
    "    hparams = tf.contrib.training.HParams(\n",
    "        learning_rate=learning_rate, dropout_rate=dropout)\n",
    "\n",
    "    summary_hook = tf.train.SummarySaverHook(\n",
    "          save_steps = run_config.save_summary_steps,\n",
    "          scaffold= tf.train.Scaffold(),\n",
    "          summary_op=tf.summary.merge_all())\n",
    "\n",
    "    mnist_estimator = tf.estimator.Estimator(\n",
    "        model_fn=model_fn,\n",
    "        config=run_config,\n",
    "        params=hparams\n",
    "    )\n",
    "\n",
    "\n",
    "    train_input_fn = data_input_fn(train_filenames[0], batch_size=batch_size)\n",
    "    eval_input_fn = data_input_fn(validation_filenames[0], batch_size=batch_size)\n",
    "\n",
    "    experiment = tf.contrib.learn.Experiment(\n",
    "        mnist_estimator,\n",
    "        train_input_fn=train_input_fn,\n",
    "        eval_input_fn=eval_input_fn,\n",
    "        train_steps=num_steps,\n",
    "        min_eval_frequency=5,\n",
    "        eval_hooks=[summary_hook]\n",
    "    )\n",
    "\n",
    "    experiment.train_and_evaluate()\n",
    "    \n",
    "    accuracy_score = mnist_estimator.evaluate(input_fn=eval_input_fn, steps=num_steps)[\"accuracy\"]\n",
    "    \n",
    "    return accuracy_score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter search <a class=\"anchor\" id='hyperparam'></a>\n",
    "\n",
    "Hyperparameter optimization is critical to achieve the best accuracy for your model. With Hops, hyperparameter optimization is easier than ever.  We can find the best hyperparameters to train the model and make it easy to find the best set of hyper parameters by visualizing them in TensorBoard for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Define dict for hyperparameters\n",
    "search_dict = {'learning_rate': [0.001, 0.0001], 'dropout': [0.45, 0.7]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hops import experiment\n",
    "from hops import hdfs\n",
    "\n",
    "experiment.differential_evolution(wrapper, search_dict, direction='max', generations=4, population=5,\n",
    "                       name='fashion mnist grid search', \n",
    "                       description='Demonstration of running gridsearch hyperparameter optimization with fashion mnist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitoring execution - TensorBoard <a class=\"anchor\" id='tensorboard'></a>\n",
    "To find the TensorBoard for the execution, please go back to HopsWorks and enter the Experiments service.\n",
    "Then copy & paste the experiment_id into the textbox and press enter to start a TensorBoard to see all experiments being run in parallel.\n",
    "![Image7-Monitor.png](../../../images/experiments_service.png)\n",
    "![Image7-Monitor.png](../../../images/tensorboard.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PySpark",
   "language": "",
   "name": "pysparkkernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 2
   },
   "mimetype": "text/x-python",
   "name": "pyspark",
   "pygments_lexer": "python2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}